Push-SAGA: A Decentralized Stochastic Algorithm With Variance Reduction Over Directed Graphs

In this letter, we propose Push-SAGA, a decentralized stochastic first-order method for finite-sum minimization over directed network of nodes. Push-SAGA combines node-level variance reduction to remove the uncertainty caused by gradients, network-level gradient tracking address distributed nature data, and push-sum consensus tackle information exchange. We show that achieves linear convergence exact solution smooth strongly convex problems is thus first linearly-convergent algorithm arbitrary connected graphs. also characterize regime in which speed-up compared its centralized counterpart network-independent rate. illustrate behavior properties with help numerical experiments on non-convex problems.